Computational Mathematics Approaches to Deep Neural Network Architecture Design

Designing appropriate neural network architectures remains a central open challenge indeep learning, with performance depending critically on depth, width, and connectivity. Current approaches rely largely on trial and error or computationally prohibitive search algorithms. This dissertation develops principled, mathematically rigorous methods for neural network architecture adaptation, drawing on ideas from topology optimization and finite element-inspired error estimation.

 

In the first part of this presentation, we address adaptive architecture design for regression and classification tasks, tackling three fundamental questions: when to add capacity during training phase (add a new layer in a network), where to add it, and how to initialize it. A topological derivative framework quantifies the sensitivity of a shape functional to the insertion of a new layer, yielding both the optimal insertion location and a principled initialization strategy for the added layer. Complementing this, a dual weighted residual approach from finite element theory decomposes approximation error across layers to guide adaptive depth refinement, forging formal connections between neural network approximation theory and classical numerical analysis.

The second part of the presentation introduces the Linear Latent Network (LiLaN), a novel architecture designed for learning solution operators of stiff nonlinear differential equations with significant computational speedup over traditional solvers. LiLaN maps states into a la-tent space where dynamics admit closed-form solutions, eliminating numerical integration at inference time. Universal approximation guarantees are established theoretically, and numerical experiments in chemical kinetics, plasma physics, and phase-field modeling demonstrate up to three orders of magnitude speedup over traditional solvers. Together, these contributions provide a suite of computational mathematics tools for adaptive neural network architecture design and accelerated scientific machine learning.